Papers with Transformer-based neural language models
Improving BERT with Syntax-aware Local Attention (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that attention-based models benefit from more focused attention over local regions. |
| Approach: | They propose a syntax-aware local attention which restrains attention over syntactically relevant words. |
| Outcome: | The proposed model performs better on all benchmark datasets, including sentence classification and sequence labeling tasks. |
Word Order Matters When You Increase Masking (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that word order information is not always useful for pre-trained models to solve downstream tasks. |
| Approach: | They examine the effect of removing position encodings on the pre-training objective itself to see whether models can reconstruct position information from co-occurrences. |
| Outcome: | The proposed model can reconstruct position information from co-occurrences without explicit encoding, but not on some tasks. |